🧾 1. Extreme Gradient Boosting (XGBoost)

XGBoost is an open-source software library which provides a regularizing gradient boosting framework for C++, Java, Python, R, Julia, Perl, and Scala. It works on Linux, Windows, and macOS. From the project description, it aims to provide a "Scalable, Portable and Distributed Gradient Boosting Library".

Extreme Gradient Boosting Algorithm. Gradient boosting refers to a class of ensemble machine learning algorithms that can be used for classification or regression predictive modeling problems. Ensembles are constructed from decision tree models.

XGBoost is a more regularized form of Gradient Boosting. XGBoost uses advanced regularization (L1 & L2), which improves model generalization capabilities. XGBoost delivers high performance as compared to Gradient Boosting. Its training is very fast and can be parallelized across clusters.

XGBoost | Wrangling with Hyperparameters | Detailed Guide

Extreme Gradient Boosting (xgboost) is similar to gradient boosting framework but more efficient. It has both linear model solver and tree learning algorithms. So, what makes it fast is its capacity to do parallel computation on a single machine...

🛠 2. Hyperparameters

What is meant by hyperparameter tuning?

In machine learning, hyperparameter optimization or tuning is the problem of choosing a set of optimal hyperparameters for a learning algorithm. A hyperparameter is a parameter whose value is used to control the learning process. By contrast, the values of other parameters (typically node weights) are learned. The hyper-parameter tuning process is a tightrope walk to achieve a balance between underfitting and overfitting. Underfitting is when the machine learning model is unable to reduce the error for either the test or training set.

What is hyperparameter tuning example?

Some examples of model hyperparameters include: The penalty in Logistic Regression Classifier i.e. L1 or L2 regularization. The learning rate for training a neural network. The C and sigma hyperparameters for support vector machines.

Why do we use hyperparameter tuning?

Hyperparameter tuning is an essential part of controlling the behavior of a machine learning model. If we don't correctly tune our hyperparameters, our estimated model parameters produce suboptimal results, as they don't minimize the loss function. This means our model makes more errors.

⌛️ 3. XGBoost Hyperparameters for Classification

-> gbtree :

• gbtree for tree-based models;

-> gblinear :

• gblinear for linear models to run at each iteration.

-> dart :

• dart is also a tree based model.
• XGBoost mostly combines a huge number of regression trees with a small learning rate. In this situation, trees added early are significant and trees added late are unimportant. Vinayak and Gilad-Bachrach proposed a new method to add dropout techniques from the deep neural net community to boosted trees, and reported better results in some situations - it s called dart. It drops trees in order to solve the over-fitting. Trivial trees (to correct trivial errors) may be prevented. Because of the randomness introduced in the training, expect the following few differences: Training can be slower than gbtree because the random dropout prevents usage of the prediction buffer. The early stop might not be stable, due to the randomness.

It relates to which booster we are using to do boosting, commonly tree or linear model

1. booster [default=gbtree]

• gbtree for tree-based models; gblinear for linear models to run at each iteration.

2. silent [default=0]:

• To activate Silent silent mode, set it to 1,meaning off; so, no messages will be printed.

It’s generally a good idea, to keep it 0 as the messages might help in understanding the model; and how the metrics are going.

3. nthread [default to maximum number of threads available if not set]

This is used for parallel processing and number of cores in the system should be entered If you wish to run on all cores, value should not be entered and algorithm will detect automatically

depend on which booster you have chosen. We will discuss about tree based boosters here.

1. eta [default=0.3]

• Similiar to learning_rate; works with geeneralization.
• Typical final values to be used: 0.01-0.2

2. min_child_weight [default=1]

• Defines the minimum sum of weights of all observations required in a child. It is used to control over-fitting. Higher values prevent a model from learning - relations which might be highly specific to the particular sample selected for a tree.
• Too high values can lead to under-fitting hence, it should be tuned using CV.

3. max_depth [default=6]

• The maximum depth of a tree, it is also used to control over-fitting as higher depth will allow model to learn relations very specific to a particular sample. Should be tuned using CV.
• Typical values: 3-10

4. max_leaf_nodes

• The maximum number of terminal nodes or leaves in a tree.
• Since binary trees are created, a depth of ‘n’ would produce a maximum of 2^n leaves. If this is defined, GBM will ignore max_depth.

5. gamma [default=0]

• A node is split only when the resulting split gives a positive reduction in the loss function. Gamma specifies the minimum loss reduction required for a split to occur.
• Makes the algorithm conservative. The values can vary depending on the loss function and should be tuned. Can be used to control overfitting.

6. max_delta_step [default=0]

• In maximum delta step we allow each tree’s weight estimation to be. If the value is set to 0, it means there is no constraint. If it is set to a positive value, it can help making the update step more conservative.
• This is generally not used.

7. subsample [default=1]

• It denotes the fraction of observations to be randomly samples for each tree.
• Lower values make the algorithm more conservative and prevents overfitting but too small values might lead to under-fitting.
• Typical values: 0.5-1

8. colsample_bytree [default=1]

• Denotes the fraction of columns to be randomly samples for each tree. Smaller colsample_bytree povides additional regulerization.
• Typical values: 0.5-1

9. colsample_bylevel [default=1]

• Denotes the subsample ratio of columns for each split, in each level.

10. alpha [default=0]

• L1 regularization term.
• L1 regularization forces the weights of uninformative features to be zero by substracting a small amount from the weight at each iteration and thus making the weight zero, eventually. It is also called regularization for simplicity. Applies on leaf weights (rather than feature weights); larger value mean more regularization.

11. lambda [default=1]

• L2 regularization term.
• This used to handle the regularization part of XGBoost. It should be explored to reduce overfitting.
• L2 regularization acts like a force that removes a small percentage of weights at each iteration. Therefore, weights will never be equal to zero. L2 regularization penalizes (weight)² There is an additional parameter to tune the L2 regularization term which is called regularization rate (lambda).
• Smoother than alpha. Also, applies on leaf weights.

12. scale_pos_weight [default=1]

• A value greater than 0 should be used in case of high class imbalance as it helps in faster convergence.

decide on the learning scenario. For example, regression tasks may use different parameters with ranking tasks.

1. objective [default=reg:linear]

• This defines the loss function to be minimized.
• Mostly used values are:
• reg:linear - used for regressions
• reg:logistic - used for classification, when you want the decision only, not the probability.
• binary:logistic –logistic regression for binary classification, returns predicted probability (rather than decision class)
• multi:softmax –multiclass classification using the softmax objective, returns predicted class (not probabilities); you also need to set an additional num_class (number of classes) parameter defining the number of unique classes
• multi:softprob –same as softmax, but returns predicted probability of each data point belonging to each class.

2. eval_metric [ default according to objective ]

• The metric to be used for validation data.
• The default values are rmse for regression and error for classification.
• Typical values are:
• rmse – root mean square error
• mae – mean absolute error
• logloss – negative log-likelihood
• error – Binary classification error rate (0.5 threshold)
• merror – Multiclass classification error rate
• mlogloss – Multiclass logloss
• auc – Area under the curve

3. seed [default=0]

• for reproducibility.

• relate to behavior of CLI version of XGBoost.

🧮 4. Understanding Bias-Variance Tradeoff

If you take a machine learning or statistics course, this is likely to be one of the most important concepts. When we allow the model to get more complicated (e.g. more depth), the model has better ability to fit the training data, resulting in a less biased model. However, such complicated model requires more data to fit.

Most of parameters in XGBoost are about bias variance tradeoff. The best model should trade the model complexity with its predictive power carefully. Parameters Documentation will tell you whether each parameter will make the model more conservative or not. This can be used to help you turn the knob between complicated model and simple model.

More Resources :

• 📋 Bias-Variance Tradeoff ➡️ with NumPy & Seaborn
• Mastering Bias-Variance Tradeoff wih Polynomials

📑 6. Best approch to Hyper parameter tuning

Base parameters :

• booster = "gbtree", as it is a classification problem
• objective = "binary:logistic", as per the problem
• tree_method="gpu_hist" using GPU

Pointers:

• Taking num_boosting_rounds = 10; to avoid overfitting

Pointers:

• A good starting Classifier, with train-auc-mean of 0.900422, test-auc-mean of 0.842716. Let;s keep tuning and reduce overfitting.

Tips: Keep it around 3-10.

Pointers:

• Taking max_depth 5, as per the score.

Tips: Keep it small for imbalanced datasets,good for balanced

Pointers:

• Taking min_child_weight 1, as per the score.

Tips: Keep it small like 0.1-0.2 forstarting. Will be tuned later.

Pointers:

• Taking gamma t 1, as per the score.

Tips: Keep it small in range 0.5-0.9.

Pointers:

• Taking 0.7

Tips: Keep it small in range 0.5-0.9.

Pointers:

• Taking 0.8

Tips: Based on class imbalance.

Pointers:

• Taking 1

Tips: Based on class imbalance.

Pointers:

• Taking .15

Tips: Based on class imbalance.

Pointers:

• Taking 1

Pointers:

• Taking .3

📒 5. Which parameter does what?

When you observe high training accuracy, but low test accuracy, it is likely that you encountered overfitting problem. There are in general two ways that you can control overfitting in XGBoost:

• The first way is to directly control model complexity.
  • This includes max_depth, min_child_weight and gamma.
• The second way is to add randomness to make training robust to noise.
  • This includes subsample and colsample_bytree.
  • You can also reduce stepsize eta. Remember to increase num_round when you do so.

Normal Condition

So, the difference between train and test AUC score is about 0.05, aka 5% - which is quite higher.

Let's regularise.

Overfitting Controlled

So, the difference between train and test AUC score is less than 0.01, aka 1% - which is quite better.

So, now the difference between train and test AUC score is less than 0.035, aka 3.5% - which is better than normal.

There’s a parameter called tree_method, set it to hist or gpu_hist for faster computation.

For common cases such as ads clickthrough log, the dataset is extremely imbalanced. This can affect the training of XGBoost model, and there are two ways to improve it.

• If you care only about the overall performance metric (AUC) of your prediction
  • Balance the positive and negative weights via scale_pos_weight
  • Use AUC for evaluation
• If you care about predicting the right probability
  • In such a case, you cannot re-balance the dataset
  • Set parameter max_delta_step to a finite number (say 1) to help convergence

📄 References

• XGBoost tips & tricks
• Binary Classification: XGBoost Hyperparameter Tuning Scenarios
• Complete Guide to Parameter Tuning in XGBoost with codes in Python - Aarshay Jain
• Notes on Parameter Tuning
• XGBoost Parameters
• L1 and L2 Regularization — Explained | by Soner Yıldırım
• XGBoost | Wrestling with Hyperparameters